Physical Characteristics. Baker County is on the Florida-Georgia border between Nassau, Duval, and Clay counties to the east and Columbia County to the west. To the south are Union and Bradford counties. The county has 585.17 square miles of land and 3.40 square miles of water for a total of 588.57 square miles. The average January temperature is 55.0 degrees Fahrenheit, and the average August temperature is 81.2 degrees Fahrenheit. The average annual rainfall is 49.27 inches. History. Baker County was established in 1861 from New River County and named for James McNair Baker (1822-92), Confederate States senator and judge in Florida's Fourth Judicial District. The Civil War Battle of Olustee was fought just east of the town of Olustee.